1. Field of the Invention
The present invention relates to a technology in which additional information such as copyright information and editing information can be embedded in or read from data such as a picture. The present invention relates to, for example, a digital watermark embedding device and method that execute a process (called “digital watermarking” or “data hiding”) of embedding a digital watermark in a picture as additional information that cannot be perceived when the picture is observed normally.
2. Description of the Related Art
With advances in digital technology, digital recording/playback devices that eliminate problems caused by repeatedly executing playback processing, such as picture quality deterioration and sound quality deterioration, have come into widespread use. In addition, various types of digital content such as various pictures and pieces of music have become distributable by using media such as digital versatile disks (DVDs) and compact disks (CDs) or by networks.
Unlike analog recording and playback, in digital recording/playback technology, quality similar to that of the original data is maintained because the data does not deteriorate even if the data is repeatedly recorded or played back. Widespread use of this digital recording/playback technology results in a flood of unauthorized copies, which is a big problem from the point of view of copyright protection.
To cope with copyright infringement caused by unauthorized copies of digital content, a system for preventing unauthorized copying is known. This system functions by adding copy control information for controlling copying of digital content, reading the copy control information in a content recording or playback mode, and executing processing in accordance with the read copy control information.
There are various systems for controlling copying of content. For example, among them, a common one is the Copy Generation Management System (CGMS). When the CGMS is applied to analog video signals (which may be called “CGMS-A”), among 20 bits of additional information to be superimposed on an effective video part in one specified horizontal interval in the vertical blanking period of the brightness signal, for example, on an effective video part in the twentieth horizontal interval in the case of an NTSC (National Television System Committee) signal, two bits are superimposed as copy control information. When the CGMS is applied to digital video signals (which may be called “CGMS-D”), the signals are transmitted in a form including 2-bit copy-control information as additional information to be added to digital video data.
In the CGMS, the 2-bit information (hereinafter referred to as the “CGMS information”) has the following meanings: “00” indicates that content may be copied; “10” indicates that content may be copied once (copying is permitted for only one generation); and “11” indicates that content is prohibited from being copied (strict prohibition of copying).
The above CGMS is one type of common copy control system. In addition, there are other systems for protecting the copyright of content. For example, digital broadcasting by broadcasting stations employs a copy generation control system that, by storing a digital copy control descriptor in program information (i.e., service information) included in transport stream (TS) packets constituting the digital data, performs copy generation control in accordance with the digital copy control descriptor when data received by a receiver is recorded in a recording unit.
Since the above descriptor is added as bit data to, for example, the content header, it is impossible to completely exclude the possibility of interpolation of the added data. A system that is advantageous in excluding the possibility of data interpolation is digital watermarking. It is impossible to view or perceive a watermark under normal playback conditions of the content (picture data or audio data). Embedding and detection of the watermark can be performed only by executing a particular algorithm or by a particular device. When content is processed by a device such as a receiver or a recording/playback unit, by detecting the watermark and controlling the processing in accordance with the watermark, reliable control can be realized.
In the method that employs digital watermarking to protect of the copyright of content, etc., which is transmitted by using a recording medium such as an optical disk, or by media such as satellite, terrestrial waves, or cable from a broadcasting station or the like, signals generated by modulating copyright data is recorded in a form in which they can be superimposed on video signals or audio signals. In the digital network age in which various types of digital content, such as pictures, sound, and data, might be copied and distributed without deteriorating, watermarking is a major technology of protecting copyright by embedding information in content itself.
Information that can be embedded by watermarking includes, not only the above copy control information, but also various types of information such as content copyright information, content modification information, content structure information, content processing information, content editing information, and content-playback-system information. For example, by using watermarks to embed pieces of editing information in a content editing mode, recognition of each process step can be performed in each editing step by referring to its watermark. This editing information is embedded as a new watermark in the content, for example, in each step of editing the content, and a final process such as removal of the watermark from the content is performed.
Various techniques for embedding and detecting digital watermarks in data have been proposed. A common digital watermarking technique is a technique based on data representing original signals, for example, statistical properties of pictures. Based on the statistical properties of picture signals such as digital video signals, a method that embeds a digital watermark while using pseudorandom-noise-series (PN-series) random-number data as a basic pattern is described below. For brevity of description, it is assumed that brightness-signal frame data has a horizontal size of eight pixels and a vertical size of six pixels.
First, PN-series random-number data PN is set as follows:
                    WN        =                  PN          =                      (                                                                                +                    1                                                                                        -                    1                                                                                        +                    1                                                                                        +                    1                                                                                        -                    1                                                                                        +                    1                                                                                        -                    1                                                                                        -                    1                                                                                                                    +                    1                                                                                        +                    1                                                                                        -                    1                                                                                        -                    1                                                                                        -                    1                                                                                        +                    1                                                                                        -                    1                                                                                        +                    1                                                                                                                    -                    1                                                                                        +                    1                                                                                        +                    1                                                                                        -                    1                                                                                        -                    1                                                                                        +                    1                                                                                        -                    1                                                                                        +                    1                                                                                                                    +                    1                                                                                        -                    1                                                                                        -                    1                                                                                        -                    1                                                                                        +                    1                                                                                        -                    1                                                                                        -                    1                                                                                        -                    1                                                                                                                    -                    1                                                                                        -                    1                                                                                        +                    1                                                                                        +                    1                                                                                        +                    1                                                                                        -                    1                                                                                        -                    1                                                                                        +                    1                                                                                                                    +                    1                                                                                        +                    1                                                                                        -                    1                                                                                        +                    1                                                                                        -                    1                                                                                        -                    1                                                                                        +                    1                                                                                        -                    1                                                                        )                                              (        1        )            
The PN-series random-number data PN is generated so that its sum is statistically zero. Next, embedding information DC is processed by performing a spread spectrum method, using the random-number data PN having the properties indicated by expression (1). In other words, when the polarity of the embedding information DC is “1”, by using the pattern of the random-number data PN without changing it, a digital watermark pattern WM is expressed as follows:
                    PN        =                  (                                                                      +                  1                                                                              -                  1                                                                              +                  1                                                                              +                  1                                                                              -                  1                                                                              +                  1                                                                              -                  1                                                                              -                  1                                                                                                      +                  1                                                                              +                  1                                                                              -                  1                                                                              -                  1                                                                              -                  1                                                                              +                  1                                                                              -                  1                                                                              +                  1                                                                                                      -                  1                                                                              +                  1                                                                              +                  1                                                                              -                  1                                                                              -                  1                                                                              +                  1                                                                              -                  1                                                                              +                  1                                                                                                      +                  1                                                                              -                  1                                                                              -                  1                                                                              -                  1                                                                              +                  1                                                                              -                  1                                                                              -                  1                                                                              -                  1                                                                                                      -                  1                                                                              -                  1                                                                              +                  1                                                                              +                  1                                                                              +                  1                                                                              -                  1                                                                              -                  1                                                                              +                  1                                                                                                      +                  1                                                                              +                  1                                                                              -                  1                                                                              +                  1                                                                              -                  1                                                                              -                  1                                                                              +                  1                                                                              -                  1                                                              )                                    (        2        )            
When the polarity of the embedding information DC is “0”, by using a pattern in which the random-number data PN is inverted, the digital watermark pattern WM is expressed as follows:
                    WN        =                              -            PN                    =                      (                                                                                -                    1                                                                                        +                    1                                                                                        -                    1                                                                                        -                    1                                                                                        +                    1                                                                                        -                    1                                                                                        +                    1                                                                                        +                    1                                                                                                                    -                    1                                                                                        -                    1                                                                                        +                    1                                                                                        +                    1                                                                                        +                    1                                                                                        -                    1                                                                                        +                    1                                                                                        -                    1                                                                                                                    +                    1                                                                                        -                    1                                                                                        -                    1                                                                                        +                    1                                                                                        +                    1                                                                                        -                    1                                                                                        +                    1                                                                                        -                    1                                                                                                                    -                    1                                                                                        +                    1                                                                                        +                    1                                                                                        +                    1                                                                                        -                    1                                                                                        +                    1                                                                                        +                    1                                                                                        +                    1                                                                                                                    +                    1                                                                                        +                    1                                                                                        -                    1                                                                                        -                    1                                                                                        -                    1                                                                                        +                    1                                                                                        +                    1                                                                                        -                    1                                                                                                                    -                    1                                                                                        -                    1                                                                                        +                    1                                                                                        -                    1                                                                                        +                    1                                                                                        +                    1                                                                                        -                    1                                                                                        +                    1                                                                        )                                              (        3        )            
When the embedding information DC consists of a plurality of information bits, by dividing, for example, the brightness-signal frame data into appropriate smaller regions, the information bits may be set to correspond to the smaller regions. Also, by using, for example, a plurality of different digital watermark patterns that are orthogonal to one another, the information bits may be set to correspond to the digital watermark patterns. Alternatively, these techniques may be used in combination.
In addition, in a picture signal such as a digital video signal, it is assumed that frame data DV1 representing a pixel level of the brightness signal in certain frame data is expressed by the following expression. In the picture signal, close brightness signals appear to have a similar pixel level, and the pixel levels of adjacent pixels are appropriately set.
                              DV          ⁢          2                =                                            DV              ⁢              1                        +            WM                    =                      (                                                            51                                                  50                                                  53                                                  55                                                  51                                                  53                                                  49                                                  48                                                                              50                                                  51                                                  50                                                  52                                                  53                                                  54                                                  49                                                  51                                                                              47                                                  51                                                  51                                                  49                                                  52                                                  53                                                  48                                                  49                                                                              50                                                  48                                                  49                                                  47                                                  50                                                  51                                                  49                                                  48                                                                              47                                                  47                                                  51                                                  50                                                  48                                                  49                                                  51                                                  51                                                                              50                                                  51                                                  51                                                  52                                                  50                                                  51                                                  56                                                  52                                                      )                                              (        4        )            
Digital watermark embedding is implemented by adding the digital watermark pattern WM to frame data DV1 of the brightness signal. When the polarity of the digital embedding information DC is “1”, the digital watermark pattern WM expressed by expression (2) is added to the brightness signal expressed by expression (4). Frame data DV2 in which the digital watermark pattern WM is imbedded is expressed as follows:
                              DV          ⁢          1                =                  (                                                    50                                            51                                            52                                            54                                            52                                            52                                            50                                            49                                                                    49                                            50                                            51                                            53                                            54                                            53                                            50                                            50                                                                    48                                            50                                            50                                            50                                            51                                            52                                            49                                            48                                                                    49                                            49                                            50                                            48                                            49                                            50                                            50                                            49                                                                    48                                            48                                            50                                            49                                            47                                            50                                            52                                            50                                                                    49                                            50                                            52                                            51                                            51                                            52                                            55                                            53                                              )                                    (        5        )            
In order to detect embedding information DC from frame data DV2 of the brightness signal in which the digital watermark pattern WM is embedded, PN-series random-number data PN identical to that used for embedding is used. The inner product P1 of original frame data DV1 of the brightness signal and random-number data PN has a value expressed as follows:P1=DV1·PN=1  (6)
Based on the statistical properties of the picture signal, the inner product P1 is a value close to zero. The inner product P2 of the frame data DV2 of the brightness signal in which digital watermark pattern WM is embedded and random-number data PN has, when the polarity of embedding information DC is “1” a value expressed as follows:
                                                                                             P                  ⁢                  2                                =                                                      DV                    ⁢                    2                                    ·                  PN                                                                                                        =                                                      (                                                                  DV                        ⁢                        1                                            +                      WM                                        )                                    ·                  PN                                                                                                        =                                                      (                                                                  DV                        ⁢                        1                                            +                      PN                                        )                                    ·                  PN                                                                                                        =                                                      P                    ⁢                    1                                    +                                      PN                    ⁢                    2                                                                                                                          =                                  1                  +                  48                                                                                          (          7          )                    
When the polarity of the embedding information DC is “0”, the inner product P2 has a value expressed as follows:
                                                                                             P                  ⁢                  2                                =                                                      DV                    ⁢                    2                                    ·                  PN                                                                                                        =                                                      (                                                                  DV                        ⁢                        1                                            +                      WM                                        )                                    ·                  PN                                                                                                        =                                                      (                                                                  DV                        ⁢                        1                                            -                      PN                                        )                                    ·                  PN                                                                                                        =                                                      P                    ⁢                    1                                    -                                      PN                    ⁢                    2                                                                                                                          =                                  1                  -                  48                                                                                          (          8          )                    
In other words, the absolute value of the inner product P2 is a value close to the random-number data PN itself. For each of several pictures, by calculating the inner product P1 of original frame data DV1 of the brightness signal and random-number data PN, and the inner product P2 of the frame data DV2 of the brightness signal in which the digital watermark pattern WM is embedded and random-number data PN, the distributions of inner products P1 and P2 can be represented by the probability density function shown in FIG. 1. Accordingly, by setting an appropriate threshold value TH, which is not negative, it can be determined whether or not digital watermark information is embedded as the digital watermark pattern WM, and it can be determined whether the polarity of the information is positive or negative, as shown below:When P2≦−TH, watermark information is embedded (polarity “0”).When |P2|<TH, no watermark information is embedded.When P2≦TH, watermark information is embedded (polarity “1”).  (9)
As the above expression (9) shows, the embedded information DC can be detected from frame data D2 of the brightness signal in which digital watermark pattern WM is embedded.
For the actual implementation of digital watermarking, two points are important: one is reliability of the digital watermark detection and the other is the influence on the picture quality of the digital watermark information. In order to accurately determine whether or not watermark information is embedded, a threshold value TH must be set so that, in FIG. 1, the probability density function when the digital watermark pattern WM is embedded and the probability density function when the digital watermark pattern WM is not embedded can be separated with good precision. However, actually, the skirts of both functions overlap with each other, so that it is difficult to select a threshold value TH that enables determination of whether or not the digital watermark pattern WM is embedded. The probability that it may be determined that digital watermark information is embedded, even though the information is not embedded, is called a “false positive value”. To guarantee secure distribution of content, an extremely small false positive value is required. Accordingly, to increase the reliability of digital watermark detection, a process that uses nonnegative scalar value C to increase the intensity of the embedded digital watermark information is executed. Frame data DV2 of the brightness signal generated when digital watermark information is embedded using the embedding intensity increased by using scalar value C has a value expressed as follows:DV2=DV1+CWM  (10)
The inner product P2 of the watermark-embedded frame data DV2 and the random-number data PN may be sufficiently increased. Specifically, the watermark-embedded frame data DV2 has a value expressed as follows:
                                                                                             P                  ⁢                  2                                =                                                      DV                    ⁢                    2                                    ·                  PN                                                                                                        =                                                      (                                                                  DV                        ⁢                        1                                            +                      CWM                                        )                                    ·                  PN                                                                                                        =                                                      (                                                                  DV                        ⁢                        1                                            ±                      CPN                                        )                                    ·                  PN                                                                                                        =                                                      P                    ⁢                    1                                    ±                                      CPN                    ⁢                    2                                                                                                            (          11          )                    
When the embedding intensity is increased as described above, the influence on the digital watermark picture quality cannot be ignored. The reliability of digital watermark detection and the influence on the picture quality of the watermark information are in a trade-off relationship.
The embedded digital watermark information must be correctly detected even if a digital-watermark-embedded picture is attacked by various techniques such as picture-format conversion, digital-to-analog conversion, MPEG compression, filtering, clipping, resizing, and rotation. There is a possibility that those who unlawfully steal copyright may maliciously attack digital-watermark-embedded pictures. Accordingly, various techniques have been proposed to enhance the resistance to these attacks, and to ensure the reliability of digital watermark detection. Nevertheless, a digital watermarking technology having strong resistance to all types of attacks has not been developed yet, and prompt measures are required.
FIG. 2 illustrates embedding of copy control information by using digital watermark information and copy control based on digital watermark detection.
When a digital video signal DV as a video source is transmitted from a broadcasting station or the like by using a medium such as satellite, terrestrial waves, or cable, a digital watermark embedding device 1 adds digital watermark information DC to the digital video signal DV, and transmits the obtained signal. The digital watermark information DC to be embedded includes video-source copyright information, and “Copy Once” information indicating that content may be copied once.
When the digital video signal DV that is transmitted after the “Copy Once” information (digital watermark information DC) is embedded therein by digital watermarking is received and played back by a set-top box or the like, the embedded digital watermark information DC is detected from the playback video signal by a digital watermark detecting device 3 built into a recording apparatus 2.
When the detecting device 3 in the recording apparatus 2 detects the “Copy Once” information, the digital watermark information DC is rewritten and recorded on an optical disk 5 by a digital watermark rewriting device 4 built into the recording apparatus 4. The rewritten digital watermark information DC includes video-source copyright information, and “No More Copy” information indicating that no further copies of the content can be made.
When the optical disk 5 on which a copy of the digital video signal DV is recorded as described above is provided, for example, as a commercially distributed medium, if someone attempts to copy the content of the provided optical disk 5 by a recording apparatus 6 again, another digital watermark detecting device 3 built into a recording apparatus 6 detects the embedded digital watermark information DC from the played back digital video signal DV.
Since the detected digital watermark information DC is the “No More Copy” information, indicating that no more copies of the content may be made, the recording apparatus 6 performs a process in accordance with the “No More Copy” information, that is, it stops recording the digital video signal DV on the optical disk 7. This is used for copy generation management.
Also, when the digital watermark embedding device 1 produces an optical disk 8 storing the digital video signal DV as a video source, it can execute the process of recording content to the optical disk 8 after embedding, in the digital video signal DV, digital watermark information DC having “Never Copy” information indicating that content is prohibited from being copied.
When the optical disk 8 on which the digital video signal DV is recorded is commercially distributed, if someone attempts to play back the content of the distributed optical disk 8 and copy it to another recording medium, the digital watermark detecting device 3 in the recording apparatus 6 detects the embedded digital watermark information DC from the played back digital video signal DV.
Since the detected watermark information DC is the “Never Copy” information indicating that content is prohibited from being copied, the recording apparatus 6 performs a process in accordance with the “Never Copy” information, that is, it stops recording to the optical disk 8. This prevents unauthorized copying.
In order to prevent a digital watermark from affecting the picture quality while ensuring the reliability of watermark detection, a digital watermark embedding technique that effectively uses human visual characteristics has been proposed. However, a method adapted for superimposing a watermark pattern on a picture while following the motion of the picture has not yet been implemented.